Effects of integration of facial expression and emotional voice on inhibition of return

doi:10.3724/SP.J.1041.2022.00331

Abstract

Abstract:

Inhibition of return (IOR) and emotion have the characteristics of guiding attentional bias and improving search efficiency. However, it is not clear whether there is a certain interaction between IOR and emotional stimuli. The study adopted the cue-target paradigm and used audiovisual dual modality to present emotional stimuli to further investigate the interaction between emotion and IOR. In Experiment 1, emotional stimuli were presented in visual single modality or audiovisual dual modality. Experiment 2 further investigated whether the impact of the audiovisual dual modality emotional stimulus on IOR was caused by the emotional stimulus of the auditory modality, that is, whether the emotional stimulus of the auditory modality was processed. The results showed that congruently emotional stimuli in the audiovisual dual modality can weaken IOR, but there was no interaction between incongruent emotional stimuli in the audiovisual dual modality and IOR, and there was no significant difference in the IOR effect between the single modality and audiovisual dual modality. The results showed that the IOR effect was influenced only when the audiovisual dual modality presented the same emotion, which further supported the perceptual inhibition theory of IOR.

Key words: inhibition of return, emotion, audiovisual integration

ZHANG Ming, WANG Tingting, WU Xiaogang, ZHANG Yue’e, WANG Aijun. (2022). Effects of integration of facial expression and emotional voice on inhibition of return. Acta Psychologica Sinica, 54(4), 331-342.

Figures/Tables 5

Figure 1. Schematic diagram of the experiment 1 procedure.

Figure 2. Reaction time under various conditions (Note. *p < 0.05, **p < 0.01)

Figure 3. (a) The amount of IOR effect under each condition (RT cued-RT uncued), (b) Relative multisensory response enhancement (rMRE) under each condition (Note. *p < 0.05, **p < 0.01, ***p < 0.001)

Figure 4. Reaction time under various conditions (Note. n.s. p > 0.05)

Figure 5. (a) The magnitude of the IOR effect under each condition (RT _cued-RT _uncued), (b) Relative multisensory response enhancement (rMRE) under each condition (Note. n.s. p > 0.05)

References 54

[1]	Anderson, A. K., Christoff, K., Panitz, D., de Rosa, E., & Gabrieli, J. D. E. (2003). Neural correlates of the automatic processing of threat facial signals. The Journal of Neuroscience, 23(13), 5627-5633. doi: 10.1523/JNEUROSCI.23-13-05627.2003 URL
[2]	Anderson, A. K., & Phelps, E. A. (2001). Lesions of the human amygdala impair enhanced perception of emotionally salient events. Nature, 411(6835), 305-309. doi: 10.1038/35077083 URL
[3]	Baijal, S., & Srinivasan, N. (2011). Emotional and hemispheric asymmetries in shifts of attention: An ERP study. Cognition and Emotion, 25(2), 280-294. doi: 10.1080/02699931.2010.492719 URL
[4]	Belin, P., Fillion-Bilodeau, S., & Gosselin, F. (2008). The Montreal Affective Voices: A validated set of nonverbal affect bursts for research on auditory affective processing. Behavior Research Methods, 40(2), 531-539. doi: 10.3758/BRM.40.2.531 URL
[5]	Berdica, E., Gerdes, A. B. M., & Alpers, G. W. (2017). A comprehensive look at phobic fear in inhibition of return: Phobia-related spiders as cues and targets. Journal of Behavior Therapy and Experimental Psychiatry, 54, 158-164. doi: S0005-7916(16)30032-5 pmid: 27517673
[6]	Chen, Q., Wei, P., & Zhou, X. (2006). Distinct neural correlates for resolving stroop conflict at inhibited and noninhibited locations in inhibition of return. Journal of Cognitive Neuroscience, 18(11), 1937-1946. pmid: 17069483
[7]	Collignon, O., Girard, S., Gosselin, F., Roy, S., Saint-Amour, D., Lassonde, M., & Lepore, F. (2008). Audiovisual integration of emotion expression. Brain Research, 1242, 126-135. doi: 10.1016/j.brainres.2008.04.023 pmid: 18495094
[8]	de Gelder, B., & Vroomen, J. (2000). The perception of emotions by ear and by eye. Cognition and Emotion, 14(3), 289-311. doi: 10.1080/026999300378824 URL
[9]	Faul, F., Erdfelder, E., Buchner, A., & Lang, A. G. (2009). Statistical power analyses using GPower 3.1: Tests for correlation and regression analyses. Behavior Research Methods*, 41(4), 1149-1160. doi: 10.3758/BRM.41.4.1149 URL
[10]	Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). GPower 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods*, 39(2), 175-191. doi: 10.3758/BF03193146 URL
[11]	Fox, E., Russo, R., & Dutton, K. (2002). Attentional bias for threat: Evidence for delayed disengagement from emotional faces. Cognition and Emotion, 16(3), 355-379. doi: 10.1080/02699930143000527 URL
[12]	Guan, J., Li, W. R., & Zhao, X. D. (2018). The competition between inhibition of return and emotional attention bias: Evidence from eye movements. Journal of Psychological Science, 41(06), 1353-1358.
[13]	Hopfinger, J. B., & Mangun, G. R. (2001). Tracking the influence of reflexive attention on sensory and cognitive processing. Cognitive, Affective, & Behavioral Neuroscience, 1(1), 56-65.
[14]	Jessen, S., & Kotz, S. A. (2011). The temporal dynamics of processing emotions from vocal, facial, and bodily expressions. NeuroImage, 58(2), 665-674. doi: 10.1016/j.neuroimage.2011.06.035 pmid: 21718792
[15]	Jia, L., Wang, J., Zhang, K., Ma, H., & Sun, H. J. (2019). Do emotional faces affect inhibition of return? An ERP study. Frontiers in Psychology, 10(721), 1-8. doi: 10.3389/fpsyg.2019.00001 URL
[16]	Klasen, M., Kreifelts, B., Chen, Y. H., Seubert, J., & Mathiak, K. (2014). Neural processing of emotion in multimodal settings. Frontiers in Human Neuroscience, 8(822), 1-4.
[17]	Klein, R. M. (2000). Inhibition of return. Trends in Cognitive Sciences, 4(4), 138-147. doi: 10.1016/S1364-6613(00)01452-2 URL
[18]	Kokinous, J., Kotz, S. A., Tavano, A., & Schröger, E. (2014). The role of emotion in dynamic audiovisual integration of faces and voices. Social Cognitive and Affective Neuroscience, 10(5), 713-720. doi: 10.1093/scan/nsu105 URL
[19]	Lang, P. J., Davis, M., & Öhman, A. (2000). Fear and anxiety: Animal models and human cognitive psychophysiology. Journal of Affective Disorders, 61(3), 137-159. pmid: 11163418
[20]	Lange, W. G., Heuer, K., Reinecke, A., Becker, E. S., & Rinck, M. (2008). Inhibition of return is unimpressed by emotional cues. Cognition and Emotion, 22(8), 1433-1456. doi: 10.1080/02699930701809451 URL
[21]	Li, P., Zhang, M. M., Li, S. X., Zhang, H.Y., & Luo, W. B. (2019). The integration of facial expression and vocal emotion and its brain mechanism. Advances in Psychological Science, 27(7), 1205-1214. doi: 10.3724/SP.J.1042.2019.01205 URL
[22]	Lupiáñez, J., Klein, R. M., & Bartolomeo, P. (2006). Inhibition of return: Twenty years after. Cognitive Neuropsychology, 23(7), 1003-1014. doi: 10.1080/02643290600588095 URL
[23]	Miller, J. (1986). Timecourse of coactivation in bimodal divided attention. Perception & Psychophysics, 40(5), 331-343. doi: 10.3758/BF03203025 URL
[24]	Mineka, S., & Öhman, A. (2002). Phobias and preparedness: The selective, automatic, and encapsulated nature of fear. Biological Psychiatry, 52(10), 927-937. pmid: 12437934
[25]	Müller, V. I., Habel, U., Derntl, B., Schneider, F., Zilles, K., Turetsky, B. I., & Eickhoff, S. B. (2011). Incongruence effects in crossmodal emotional integration. NeuroImage, 54(3), 2257-2266. doi: 10.1016/j.neuroimage.2010.10.047 URL
[26]	Okon-Singer, H., Henik, A., & Gabay, S. (2020). Increased inhibition following negative cues: A possible role for enhanced processing. Cortex, 122, 131-139. doi: S0010-9452(18)30418-0 pmid: 30638583
[27]	Pan, F., Wu, X., Zhang, L., & Ou, Y. (2017). Inhibition of return is modulated by negative stimuli: Evidence from subliminal perception. Frontiers in Psychology, 8(1012), 1-8.
[28]	Pérez-Dueñas, C., Acosta, A., & Lupiáñez, J. (2009). Attentional capture and trait anxiety: Evidence from inhibition of return. Journal of Anxiety Disorders, 23(6), 782-790. doi: 10.1016/j.janxdis.2009.03.002 pmid: 19380211
[29]	Pérez-Dueñas, C., Acosta, A., & Lupiáñez, J. (2014). Reduced habituation to angry faces: Increased attentional capture as to override inhibition of return. Psychological Research, 78(2), 196-208. doi: 10.1007/s00426-013-0493-9 pmid: 23689476
[30]	Posner, M., & Cohen, Y. (1984). Components of visual orienting. Attention and Performance X: Control of Language Processes, 32, 531-556.
[31]	Pourtois, G., de Gelder, B., Vroomen, J., Rossion, B., & Crommelinck, M. (2000). The time-course of intermodal binding between seeing and hearing affective information. NeuroReport, 11(6), 1329-1333. pmid: 10817616
[32]	Pourtois, G., Schettino, A., & Vuilleumier, P. (2013). Brain mechanisms for emotional influences on perception and attention: What is magic and what is not. Biological Psychology, 92(3), 492-512. doi: 10.1016/j.biopsycho.2012.02.007 pmid: 22373657
[33]	Prime, D. J., & Jolicœur, P. (2009). On the relationship between occipital cortex activity and inhibition of return. Psychophysiology, 46(6), 1278-1287. doi: 10.1111/psyp.2009.46.issue-6 URL
[34]	Prime, D. J., & Ward, L. M. (2006). Cortical expressions of inhibition of return. Brain Research, 1072(1), 161-174. doi: 10.1016/j.brainres.2005.11.081 URL
[35]	Redden, R. S., MacInnes, W. J., & Klein, R. M. (2021). Inhibition of return: An information processing theory of its natures and significance. Cortex, 135, 30-48. doi: 10.1016/j.cortex.2020.11.009 pmid: 33360759
[36]	Reuter-Lorenz, P. A., Jha, A. P., & Rosenquist, J. N. (1996). What is inhibited in inhibition of return. Journal of Experimental Psychology: Human Perception and Performance, 22(2), 367-378. doi: 10.1037/0096-1523.22.2.367 URL
[37]	Rutherford, H. J. V., & Raymond, J. E. (2010). Effects of spatial cues on locating emotional targets. Visual Cognition, 18(3), 389-412. doi: 10.1080/13506280902787043 URL
[38]	Silvert, L., & Funes, M. J. (2016). When do fearful faces override inhibition of return? Acta Psychologica, 163, 124-134. doi: 10.1016/j.actpsy.2015.11.002 URL
[39]	Stein, B. E., & Stanford, T. R. (2008). Multisensory integration: Current issues from the perspective of the single neuron. Nature Reviews Neuroscience, 9(4), 255-266. doi: 10.1038/nrn2331 URL
[40]	Stoyanova, R. S., Pratt, J., & Anderson, A. K. (2007). Inhibition of return to social signals of fear. Emotion, 7(1), 49-56. pmid: 17352562
[41]	Talsma, D., & Woldorff, M. G. (2005). Selective attention and multisensory integration: Multiple phases of effects on the evoked brain activity. Journal of Cognitive Neuroscience, 17(7), 1098-1114. pmid: 16102239
[42]	Tang, X., Gao, Y., Yang, W., Ren, Y., Wu, J., Zhang, M., & Wu, Q. (2019). Bimodal-divided attention attenuates visually induced inhibition of return with audiovisual targets. Experimental Brain Research, 237(4), 1093-1107. doi: 10.1007/s00221-019-05488-0 URL
[43]	Taylor, T. L., & Klein, R. M. (1998). On the causes and effects of inhibition of return. Psychonomic Bulletin & Review, 5(4), 625-643. doi: 10.3758/BF03208839 URL
[44]	Taylor, T. L., & Therrien, M. E. (2008). Inhibition of return for the discrimination of faces. Perception & Psychophysics, 70(2), 279-290. doi: 10.3758/PP.70.2.279 URL
[45]	Tian, Y., & Yao, D. (2008). A study on the neural mechanism of inhibition of return by the event-related potential in the Go/Nogo task. Biological Psychology, 79(2), 171-178. doi: 10.1016/j.biopsycho.2008.04.006 pmid: 18524452
[46]	Tottenham, N., Tanaka, J. W., Leon, A. C., McCarry, T., Nurse, M., Hare, T. A., Marcus, D. J., Westerlund, A., Casey, B. J., & Nelson, C. (2009). The NimStim set of facial expressions: Judgments from untrained research participants. Psychiatry Research, 168(3), 242-249. doi: 10.1016/j.psychres.2008.05.006 pmid: 19564050
[47]	Troiani, V., Price, E. T., & Schultz, R. T. (2014). Unseen fearful faces promote amygdala guidance of attention. Social Cognitive and Affective Neuroscience, 9(2), 133-140. doi: 10.1093/scan/nss116 URL
[48]	Ulrich, R., Miller, J., & Schröter, H. (2007). Testing the race model inequality: An algorithm and computer programs. Behavior Research Methods, 39(2), 291-302. pmid: 17695357
[49]	van der Burg, E., Talsma, D., Olivers, C. N. L., Hickey, C., & Theeuwes, J. (2011). Early multisensory interactions affect the competition among multiple visual objects. NeuroImage, 55(3), 1208-1218. doi: 10.1016/j.neuroimage.2010.12.068 URL
[50]	van der Stoep, N., van der Stigchel, S., Nijboer, T. C. W., & Spence, C. (2017). Visually induced inhibition of return affects the integration of auditory and visual information. Perception, 46(1), 6-17. doi: 10.1177/0301006616661934 pmid: 27484341
[51]	Vizueta, N., Patrick, C. J., Jiang, Y., Thomas, K. M., & He, S. (2012). Dislocational fear, negative affectivity, and neuroimaging response to visually suppressed emotional faces. NeuroImage, 59(1), 761-771. doi: 10.1016/j.neuroimage.2011.07.015 URL
[52]	Wang, J. X., Jia, L. P., Zhang, K., & Zhang, S. (2013). Influence of emotional faces for different location on inhibition of return. Journal of Psychological Science, 36(02), 335-339.
[53]	Wang, Z., Zhang, K., & Klein, R. M. (2010). Inhibition of return in static but not necessarily in dynamic search. Attention, Perception, & Psychophysics, 72(1), 76-85. doi: 10.3758/APP.72.1.76 URL
[54]	Zhao, X., Li, X., & Shi, W. (2017). Influence of inhibitory tagging (IT) on emotional and cognitive conflict processing: Evidence from event-related potentials. Neuroscience Letters, 657, 120-125. doi: 10.1016/j.neulet.2017.08.014 URL